Enabling Enterprise Responsible AI: Announcing Domino Model Sentry

Artificial intelligence technology is spreading rapidly across the economy. New use cases drive innovative new products, optimize processes and make money. Yet with AI's great power, organizations must still mitigate risk and behave responsibly.

AI risks come in many forms, including:

• Transparency: It may be impossible for humans to understand how models generate results.
• Privacy and security: Private data may leak through careless use in model training. Lax security may lead to breaches, targeted attacks, and data poisoning, diminishing a model's value. Both can severely impact a company's reputation.
• Fairness: Bias may impact models and result in outputs humans deem discriminatory.
• Safety and reliability: Insufficient testing and outdated models can result in safety and performance challenges.
• Accuracy: Models may appear to produce accurate outputs during training. They may even do so in production when they process data similar to the data used to train them. Yet real-world data may lead the model to present inaccurate and unexpected results.

The stakes have never been higher. Regulators are formulating rules and are planning on levying considerable fines. Stakeholder trust in models hinges on model quality and explainability. AI misuse can risk a company's entire business or reputation and jeopardize its future. An emerging collection of Responsible AI principles looks to help mitigate these risks. As foundations, responsible AI emphasizes improved model governance, better operational practices, and attention to human concerns.

Domino has helped customers develop and evolve their data science and AI work for much of the last decade. Over this time, the companies that thrived adopted a people and process-first approach to responsible AI. We are now taking the responsible AI lessons learned from these collaborations and announcing Domino Model Sentry.

Domino Model Sentry provides a tightly-integrated capability suite for responsibly scaling and operating AI models. Model Sentry provides enterprises with the tools to ensure accuracy, fairness, transparency, and accountability across the model lifecycle. Model Sentry spans six components:

Project Templates

Mature data science operations establish clear and detailed development and release standards. Such standards provide guardrails from pitfalls and simplify governance reviews. Model Sentry's Project Templates help define standards and best practices to include in projects to facilitate responsible AI. For example, a checklist of compliance tasks, an App that enables bias detection, etc. With templates, your team improves model governance controls without sacrificing productivity.

Model Registry

Enterprises are developing and deploying an increasing number of models to production. Domino's Model Registry helps create a secure information store to version and manage models regardless of where they were trained, all from a single pane of glass. The tool makes experiment reproduction and model recreation straightforward. As a result, teams can better assess their risk exposure, become more audit-ready, and increase their confidence when promoting a model to production.

Automated Model Cards

Model cards provide a single source of truth for all model information. Model cards automatically incorporate information about model lineage, algorithms, and experiments performed. Cards also track quality metrics, serving endpoints, model versions, and more. Cards are customizable, allowing teams to include additional model context. For Domino-hosted models, cards list the APIs, jobs, and launchers dependent on the model.

Model Review

Enterprises need to align all impacted teams before model launch. Legal, privacy, and compliance teams must review models before launch. Model Sentry's Model Review enables stakeholders across organizations and functions to operate on a single source of truth about a model. Teams can now engage via comments, review feedback, and transition a model through customizable stages.

Model Staging

Model Staging is an essential step in the model release process. Model Staging enables model deployment to separate environments. This allows organizations to roll models out in stages before having a business impact. Such testing and validation can surface various issues, including cross-system integration problems and performance bottlenecks. It can also simulate real-world processes and help staff train on new procedures.

Model Monitoring

Model operations present unique challenges, with drift being a particularly thorny issue. It is normal for model input data to evolve and result in model performance degradation. Domino Model Monitoring helps customers remain proactive and avoid nasty surprises in such circumstances. Model monitoring detects data drift and prediction accuracy. Model monitoring tracks input data and compares model outputs to ground truth. It alerts administrators if it detects significant gaps between input and training data or when output exceeds acceptable performance thresholds. These alerts allow customers to respond to risk by disabling or retraining the model.

***

The rapid expansion of AI presents numerous benefits. But companies must also responsibly manage the risks associated with AI. Domino's Model Sentry helps companies adopt a people and process-based approach to responsible AI. Model Sentry helps companies manage their AI lifecycle. Furthermore, it enhances audit readiness and eases navigating the technology's complex landscape.

We look forward to helping companies adopt the technology on their journey to an even better AI experience for all.